Benchmark#

Whenever we make changes to the code we will rerun a simple benchmark in order to keep track of how the simulation results might change over time. It is in general hard to verify that the equations are implemented correctly because there exist no analytic solutions to the equations. However, by running the same benchmark over time we can see if the changes we make will impact the simulation results. Here the different biomarkers are taken from traces evaluated a the center of at slab that is simulated for 1000 ms (see the benchmark in the library for more details)

from pathlib import Path
import json
import matplotlib.pyplot as plt
import numpy as np
from collections import defaultdict
import pandas as pd
import plotly.graph_objects as go
from plotly.subplots import make_subplots
import requests

pd.options.display.max_columns = 5

headers = {
    "Accept": "application/vnd.github+json",
    "X-GitHub-Api-Version": "2022-11-28",
}
print("Get existing data from gist")
response = requests.get(
    f"https://api.github.com/gists/73fa6531f28da2b3633a7ddaca38a7cd",
    headers=headers,
)
data = json.loads(response.json()["files"]["convergence_test.json"]["content"])

Get existing data from gist

You can checkout a specific commit by going to the link

ComputationalPhysiology/simcardems

where you swap out GIT_COMMIT_HASH with the specific hash, i.e

for sha in data.keys():
    print(f"https://github.com/ComputationalPhysiology/simcardems/commit/{sha}")

https://github.com/ComputationalPhysiology/simcardems/commit/a3ce91b26d63e91ce5d7701333ff3da13de38cc3
https://github.com/ComputationalPhysiology/simcardems/commit/19f9c461fbaf4206d48775c151116cff6d661369
https://github.com/ComputationalPhysiology/simcardems/commit/00dae0a1bd0b7919dee6ba4d96a4cf63f8949b0c
https://github.com/ComputationalPhysiology/simcardems/commit/7c99dbc59c2c38c7c192be00eac500546471bed0
https://github.com/ComputationalPhysiology/simcardems/commit/47edd23
https://github.com/ComputationalPhysiology/simcardems/commit/d44db58
https://github.com/ComputationalPhysiology/simcardems/commit/37fb7d1
https://github.com/ComputationalPhysiology/simcardems/commit/172470c
https://github.com/ComputationalPhysiology/simcardems/commit/5cc1a44
https://github.com/ComputationalPhysiology/simcardems/commit/7d2252c
https://github.com/ComputationalPhysiology/simcardems/commit/3b1d61b
https://github.com/ComputationalPhysiology/simcardems/commit/5992724
https://github.com/ComputationalPhysiology/simcardems/commit/5282d4f
https://github.com/ComputationalPhysiology/simcardems/commit/ce433ea
https://github.com/ComputationalPhysiology/simcardems/commit/3ed0bc0
https://github.com/ComputationalPhysiology/simcardems/commit/0e0091e
https://github.com/ComputationalPhysiology/simcardems/commit/5a86d28
https://github.com/ComputationalPhysiology/simcardems/commit/e616f0a
https://github.com/ComputationalPhysiology/simcardems/commit/5580bef
https://github.com/ComputationalPhysiology/simcardems/commit/0d80766
https://github.com/ComputationalPhysiology/simcardems/commit/7ffac90
https://github.com/ComputationalPhysiology/simcardems/commit/999fcfa
https://github.com/ComputationalPhysiology/simcardems/commit/97e958d
https://github.com/ComputationalPhysiology/simcardems/commit/4cc3e7a
https://github.com/ComputationalPhysiology/simcardems/commit/5edda13
https://github.com/ComputationalPhysiology/simcardems/commit/37f4c61
https://github.com/ComputationalPhysiology/simcardems/commit/c7df02b
https://github.com/ComputationalPhysiology/simcardems/commit/8b11e8d
https://github.com/ComputationalPhysiology/simcardems/commit/0221cc5
https://github.com/ComputationalPhysiology/simcardems/commit/6dde0c0
https://github.com/ComputationalPhysiology/simcardems/commit/a4ea024
https://github.com/ComputationalPhysiology/simcardems/commit/1f7912f
https://github.com/ComputationalPhysiology/simcardems/commit/947b107
https://github.com/ComputationalPhysiology/simcardems/commit/3d43780
https://github.com/ComputationalPhysiology/simcardems/commit/e99e183
https://github.com/ComputationalPhysiology/simcardems/commit/f4b9b88
https://github.com/ComputationalPhysiology/simcardems/commit/3a87bf1
https://github.com/ComputationalPhysiology/simcardems/commit/d5245d4
https://github.com/ComputationalPhysiology/simcardems/commit/813ba9c
https://github.com/ComputationalPhysiology/simcardems/commit/c242e33
https://github.com/ComputationalPhysiology/simcardems/commit/1ed82dc
https://github.com/ComputationalPhysiology/simcardems/commit/f5511ff
https://github.com/ComputationalPhysiology/simcardems/commit/68ae7bb
https://github.com/ComputationalPhysiology/simcardems/commit/739dc1c
https://github.com/ComputationalPhysiology/simcardems/commit/0c195ff
https://github.com/ComputationalPhysiology/simcardems/commit/71a7b0e
https://github.com/ComputationalPhysiology/simcardems/commit/9d5c1b1
https://github.com/ComputationalPhysiology/simcardems/commit/38ad52c
https://github.com/ComputationalPhysiology/simcardems/commit/9f2fd38
https://github.com/ComputationalPhysiology/simcardems/commit/c9b3528
https://github.com/ComputationalPhysiology/simcardems/commit/1353445
https://github.com/ComputationalPhysiology/simcardems/commit/df9ad53
https://github.com/ComputationalPhysiology/simcardems/commit/fbfcf55
https://github.com/ComputationalPhysiology/simcardems/commit/90b65d3
https://github.com/ComputationalPhysiology/simcardems/commit/bc47c96
https://github.com/ComputationalPhysiology/simcardems/commit/322191d
https://github.com/ComputationalPhysiology/simcardems/commit/68926df
https://github.com/ComputationalPhysiology/simcardems/commit/cbc3038
https://github.com/ComputationalPhysiology/simcardems/commit/32e43ff
https://github.com/ComputationalPhysiology/simcardems/commit/4d3a653
https://github.com/ComputationalPhysiology/simcardems/commit/09a7e5b
https://github.com/ComputationalPhysiology/simcardems/commit/9b246fe
https://github.com/ComputationalPhysiology/simcardems/commit/695cd7b
https://github.com/ComputationalPhysiology/simcardems/commit/edc973c
https://github.com/ComputationalPhysiology/simcardems/commit/f38605c
https://github.com/ComputationalPhysiology/simcardems/commit/3398fd2
https://github.com/ComputationalPhysiology/simcardems/commit/f63afc9
https://github.com/ComputationalPhysiology/simcardems/commit/8629d78
https://github.com/ComputationalPhysiology/simcardems/commit/f702955
https://github.com/ComputationalPhysiology/simcardems/commit/376167c
https://github.com/ComputationalPhysiology/simcardems/commit/69503d0
https://github.com/ComputationalPhysiology/simcardems/commit/67aba10
https://github.com/ComputationalPhysiology/simcardems/commit/cb33e06
https://github.com/ComputationalPhysiology/simcardems/commit/12ad5f2
https://github.com/ComputationalPhysiology/simcardems/commit/42363b0
https://github.com/ComputationalPhysiology/simcardems/commit/de8a7ff
https://github.com/ComputationalPhysiology/simcardems/commit/0648726
https://github.com/ComputationalPhysiology/simcardems/commit/f73b25e
https://github.com/ComputationalPhysiology/simcardems/commit/5d5bd73
https://github.com/ComputationalPhysiology/simcardems/commit/2fe1309
https://github.com/ComputationalPhysiology/simcardems/commit/04a9790
https://github.com/ComputationalPhysiology/simcardems/commit/7f51a50
https://github.com/ComputationalPhysiology/simcardems/commit/93c349e
https://github.com/ComputationalPhysiology/simcardems/commit/e8e6d35
https://github.com/ComputationalPhysiology/simcardems/commit/630ed09
https://github.com/ComputationalPhysiology/simcardems/commit/6e55904
https://github.com/ComputationalPhysiology/simcardems/commit/fde1483
https://github.com/ComputationalPhysiology/simcardems/commit/de72206
https://github.com/ComputationalPhysiology/simcardems/commit/e3254b0
https://github.com/ComputationalPhysiology/simcardems/commit/dbbec69
https://github.com/ComputationalPhysiology/simcardems/commit/f214f80
https://github.com/ComputationalPhysiology/simcardems/commit/159c6bc
https://github.com/ComputationalPhysiology/simcardems/commit/a3a23b2
https://github.com/ComputationalPhysiology/simcardems/commit/60e69df
https://github.com/ComputationalPhysiology/simcardems/commit/cf7cc9d
https://github.com/ComputationalPhysiology/simcardems/commit/10592e9
https://github.com/ComputationalPhysiology/simcardems/commit/9d72060
https://github.com/ComputationalPhysiology/simcardems/commit/cc84512
https://github.com/ComputationalPhysiology/simcardems/commit/e36932a

excluded_keys = ["dt", "dx", "sha", "num_cells_mechanics", "num_cells_ep", "num_vertices_mechanics", "num_vertices_ep"]
results = defaultdict(list)
for git_hash, d in data.items():
    results["git_hash"].append(git_hash)
    for key, value in d.get("dx0.2_dt0.05").items():
        if key in excluded_keys:
            continue
        results[key].append(value)
        
for k, v in results.items():
    print(k, len(v))

git_hash 99
import_time 99
timestamp 99
simcardems_version 99
create_runner_time 99
solve_time 99
maxTa 99
ampTa 99
APD40 99
APD50 99
APD90 99
triangulation 99
Vpeak 99
Vmin 99
dvdt 99
maxCa 99
ampCa 99
CaTD50 99
CaTD80 99
CaTD90 99
ttp_Ta 99
rt50_Ta 99
rt95_Ta 99
maxlmbda 99
minlmbda 99
ttplmbda 99
lmbdaD50 99
lmbdaD80 99
lmbdaD90 99
rt50_lmbda 99
rt95_lmbda 99
max_displacement_norm 99
rel_max_displacement_norm 99
max_displacement_perc_norm 99
rel_max_displacement_perc_norm 99
time_to_max_displacement_norm 99
max_displacement_x 99
rel_max_displacement_x 99
max_displacement_perc_x 99
rel_max_displacement_perc_x 99
time_to_max_displacement_x 99
max_displacement_y 99
rel_max_displacement_y 99
max_displacement_perc_y 99
rel_max_displacement_perc_y 99
time_to_max_displacement_y 99
max_displacement_z 99
rel_max_displacement_z 99
max_displacement_perc_z 99
rel_max_displacement_perc_z 99
time_to_max_displacement_z 99

df = pd.DataFrame(results)
df["timestamp"] = pd.to_datetime(df["timestamp"])
df
git_hash import_time ... rel_max_displacement_perc_z time_to_max_displacement_z
0 a3ce91b26d63e91ce5d7701333ff3da13de38cc3 1.200000e-06 ... 0.486886 0.05
1 19f9c461fbaf4206d48775c151116cff6d661369 9.000000e-07 ... 0.486886 0.05
2 00dae0a1bd0b7919dee6ba4d96a4cf63f8949b0c 1.100000e-06 ... 0.486886 0.05
3 7c99dbc59c2c38c7c192be00eac500546471bed0 1.300000e-06 ... 0.486886 0.05
4 47edd23 9.000000e-07 ... 0.486886 0.05
... ... ... ... ... ...
94 cf7cc9d 9.000000e-07 ... 0.249432 138.05
95 10592e9 9.000000e-07 ... 0.486886 0.05
96 9d72060 1.000000e-06 ... 0.486909 0.05
97 cc84512 8.000000e-07 ... 0.486909 0.05
98 e36932a 1.000000e-06 ... 0.486909 0.05

99 rows × 51 columns

df = df.sort_values(by='timestamp')
df
git_hash import_time ... rel_max_displacement_perc_z time_to_max_displacement_z
66 f63afc9 1.001000e-06 ... 0.249432 138.05
84 630ed09 1.000000e-06 ... 0.249432 138.05
59 4d3a653 1.100000e-06 ... 0.249432 138.05
44 0c195ff 1.000000e-06 ... 0.249432 138.05
26 c7df02b 9.000000e-07 ... 0.249432 138.05
... ... ... ... ... ...
35 f4b9b88 8.000000e-07 ... 0.486886 0.05
3 7c99dbc59c2c38c7c192be00eac500546471bed0 1.300000e-06 ... 0.486886 0.05
2 00dae0a1bd0b7919dee6ba4d96a4cf63f8949b0c 1.100000e-06 ... 0.486886 0.05
1 19f9c461fbaf4206d48775c151116cff6d661369 9.000000e-07 ... 0.486886 0.05
0 a3ce91b26d63e91ce5d7701333ff3da13de38cc3 1.200000e-06 ... 0.486886 0.05

99 rows × 51 columns

df[["simcardems_version", "timestamp", "git_hash"]]
simcardems_version timestamp git_hash
66 2022.3.0 2022-10-06 16:25:06.830650 f63afc9
84 2022.3.0 2022-10-06 16:55:04.711879 630ed09
59 2022.3.0 2022-10-06 20:09:07.230901 4d3a653
44 2022.3.0 2022-10-06 20:19:12.227464 0c195ff
26 2022.3.0 2022-10-06 20:28:25.015389 c7df02b
... ... ... ...
35 2023.3.0 2023-04-11 08:24:24.930409 f4b9b88
3 2023.3.0 2023-04-12 09:53:34.139318 7c99dbc59c2c38c7c192be00eac500546471bed0
2 2023.3.0 2023-04-12 17:17:24.396818 00dae0a1bd0b7919dee6ba4d96a4cf63f8949b0c
1 2023.3.0 2023-04-12 20:33:25.203391 19f9c461fbaf4206d48775c151116cff6d661369
0 2023.4.0 2023-05-10 07:52:46.302512 a3ce91b26d63e91ce5d7701333ff3da13de38cc3

99 rows × 3 columns

def get_ylim(values):
    if np.isclose(values, 0).all():
        return (-0.01, 0.01)
    y_mean = np.mean(values)
    y_max = max(np.max(values), y_mean + 0.01 * abs(y_mean))
    y_min = min(np.min(values), y_mean - 0.01 * abs(y_mean))
    return (y_min, y_max)

columns = [c for c in df.columns if c not in ["timestamp", "simcardems_version", "git_hash"]]
git_hash = df["git_hash"]
versions = df["simcardems_version"]
dates = [t.date() for t in df["timestamp"]]

text = []
for h,v, t in zip(git_hash, versions, dates):
    text.append("\n".join([
        f"<br>Git Hash: {h}</br>",
        f"<br>Version: {v}</br>",
        f"<br>Timestamp {t}</br>",
    ]))
    
fig = make_subplots(rows=len(columns), cols=1, subplot_titles=columns, shared_xaxes=True)

yranges = []
for i, c in enumerate(columns):
    
    row = i + 1
    col = 1
    
    fig.add_trace(
        go.Scatter(x=df["timestamp"], y=df[c], text=text, hovertemplate="%{text}"),
        row=row, col=col
    )
    fig.update_yaxes(range=get_ylim(df[c]),row=row, col=col)

fig.update_layout(
    height=5000,
    showlegend=False
)
fig.show()